High frequency batch-oriented computations over large sliding time windows

Today’s business workflows are very likely to include batch computations that periodically analyze subsets of data relative to specific time ranges in order to provide strategic information for stakeholders and other interested parties. The frequency of these computations directly impacts on how much updated such information can be, which provides an effective measure of their usefulness. This in turn drives towards solutions that allow to carry out batch elaborations more often, so as to always have updated information as soon as possible. How often they can be executed usually depends either on application-specific requirements or on some other constraints, typically about completion time because it’s uncommon to start a new batch computation before the previous one has completed, and the typical amounts of data to elaborate in these scenarios are so large that a computation can take very long. In this paper we propose a model for batch processing on sliding time window event computations that allows the definition of multiple metrics for performance optimization. These metrics specifically take into account the organization of input data to minimize its impact on computation latency. The model is then instantiated on Hadoop, a batch processing engine based on the MapReduce paradigm, and a set of strategies for efficiently arranging input data is described and evaluated.